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DETAILED ACTION 
Information Disclosure Statement 

The information disclosure statement (IDS) submitted on 8 August 2005 was filed 
after the mailing date of the previous Office Action on 4 May 2005. The submission is in 
compliance with the provisions of 37 CFR 1 .97. Accordingly, the examiner is 
considering the information disclosure statement. 

Citation AE is not in conformance with standard Office Policy. The submitted 
copy of that reference is defective. It has the phrase " http://www.pdf-tools.com " written 
in large black, bold letter diagonally across all pages that make it extremely difficult to 
read. Nonetheless, examiner has considered such a reference, but applicant is 
required to submit a clean copy under 37 CFR 1 .97/1 .98, since applicant has 
determined that such a reference could be relevant and has submitted such, examiner 
has the right to require a clean, legible copy. 

Priority 

Applicant's claim for domestic priority is denied. Examiner cannot easily 
determine which claims should receive CIP status and which consist entirely of new or 
added material that deserves of the original US filing date. As such, all claims will be 
given the actual filing date of the present US application. Applicant's lengthy 
specification and substantial filings of multiple IDSes encompassing over a hundred 
references with thousands of pages total to review have placed an undue and excessive 
burden on the examiner and the Office already. As such, applicant is required to clarify 
which claims receive which priority dates. The specification of the instant application is 
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43 pages long and has 8 pages of drawings; the parent application that it is claimed as 
a CIP of consists of a specification having 17 pages and 2 pages of drawings. 
Examiner will assume that all material added is new and only receives the US filing 
date. 

Further, the provisional application filed 15 August 2002 is not substantially 
identical to the instant application. Including drawings, it is only 13 pages long, and it is 
a conference paper, not the specification of an application. The present US case has a 
43-page specification and 8 pages of drawings for a total length of 51 pages. 
Therefore, the burden is on applicant to show that such claims in the instant application 
deserve priority from either 60/418,841 and/or 10/278,349. 

Further, such priority claims must show clearly where support is found in the 
parent application(s) for each claimed element on a column number, line number and/or 
paragraph basis. 

Response to Arguments 

Applicant's arguments, see pages 1-12 of Remarks, filed 8 August 2005, with 
respect to the rejection(s) of claim(s) 1-44 under 35 U.S.C. 103(a) have been fully 
considered and are persuasive in some respects. 

The objection to the title stands withdrawn since applicant has amended the title 
(Remarks page 1). 

Applicant's arguments that the Summary section and various other features of 
the specification are suggested but not mandatory on pages 1-2 of Remarks are found 
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to be persuasive. Therefore, the objections to the specification in paragraphs 5 and 6 of 
the previous Office Action stand withdrawn. 

The objections to the disclosure because of various informalities stand withdrawn 
because applicant has corrected these deficiencies. 

Upon further examination of the specification, particularly paragraphs [0032] and 
[0061], examiner finds that applicant's assertion that a stereo camera is not necessary 
to be correct, as stated in Remarks page 2, last four lines, and the majority of page 3. 

The rejections of claims 1-44 under 35 U.S.C. 112, first paragraph, under In re 
Mayhew, are therefore withdrawn for the above reasons. 

The rejections of claims 1 -44 under 35 U.S.C. 112, second paragraph, for the 
use of the term 'image sensor* stand withdrawn in view of applicant's clarifications as 
above, since as applicant pointed out, other image sensors apply. 

The rejections of claims 28 and 36 under 35 U.S.C. 112, second paragraph due 
to relative terminology stand withdrawn because applicant has amended the claims to 
remove the offending term (Remarks page 4). 

The rejections of various claims 24, 27, 28, 32, 35, 36, and 42-44 under 35 USC 
1 03 stand withdrawn since the claims have been canceled (Remarks page 1 ). 

The objections to claims 12, 22, 29, and 37 as being dependent upon rejected 
base claims stands withdrawn since these claims have been made independent 
(Remarks page 4). 
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However, upon further consideration, a new ground(s) of rejection is made in 
view of various references, including new ones brought up by applicant's submission of 
the IDS of 8 August 2005. 

They were rewritten in independent form, for the reasons indicated in the 
previous Office Action. 

Allowable Subject Matter 

Claims 12 and 22 are allowed for at least the reasons cited in the previous Office 
Action. They have been brought into independent form. 

Claims 29 and 37 were modified by applicant into independent format but are 
subject to an improper broadening amendment. Therefore, the allowability of such 
claims is withdrawn. See the rejection of such claims under 35 U.S.C. 112 below. 
Applicant is further put on notice that if the phrase "at least" is not deleted from the 
claims, they will be finally rejected. If applicant removes the improper broadening 
language (the "at least") the claims will be allowed. Further, applicant is put on notice 
that if such broadening language is not removed, the claims will also be rejected under 
35 U.S.C. 103(a). 

Claim Rejections - 35 USC § 112 

The following is a quotation of the first paragraph of 35 U.S.C. 112: 

The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the 
art to which it pertains, or with which it is most nearly connected, to make and use the same and shall 
set forth the best mode contemplated by the inventor of carrying out his invention. 

Claims 25-26, 29-31, 33-34, and 37-39 are rejected under 35 U.S.C. 1 12, first 
paragraph, as failing to comply with the written description requirement. The claim(s) 



Application/Control Number: 10/676,377 Page 6 

Art Unit: 2672 

contains subject matter that was not described in the specification in such a way as to 
reasonably convey to one skilled in the relevant art that the inventor(s), at the time the 
application was filed, had possession of the claimed invention. Specifically, the 
specification does not show evidence that greater than five frames were used for the 
averaging technique proposed. 

Claims 25-26, 29-31, 33-34, and 37-39 are rejected under 35 U.S.C. 1 12, first 
paragraph, as failing to comply with the enablement requirement. The claim(s) contains 
subject matter which was not described in the specification in such a way as to enable 
one skilled in the art to which it pertains, or with which it is most nearly connected, to 
make and/or use the invention. Specifically, the specification is not enabling for more 
than five frames being used in the averaging technique. Furthermore, extrinsic 
evidence clearly shows that the use of various numbers of frames in the averaging 
technique can produce very different results. Examiner will provide numerous citations 
to this effect if requested to do so. A further, more technical explanation will be 
provided as well in the next response should applicant so request. 

Examiner contends that applicant has established criticality for the use of five 
frames in the averaging technique, which therefore would render the use of "more than 
five" frames into an area where the results would be at best unpredictable and at worst 
undesirable. Further, optimizing such a number of frames would require undue 
experimentation under In re Wands, since applicant points out that five frames has a 
specific benefit, as per the specification. A fuller, more complete and detailed response 
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entailing all of the In re Wands factors will also be made in the next response should 
applicant fail to remove the broadening language. 

Specifically, claims 29 and 37 have these defects, and claims 25-26, 30-31 , 33- 
34, and 38-39 are dependent upon them but fail to correct the deficiencies of their 
parent claim(s). 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as 
set forth in section 102 of this title, if the differences between the subject matter sought to be 
patented and the prior art are such that the subject matter as a whole would have been obvious 
at the time the invention was made to a person having ordinary skill in the art to which said 
subject matter pertains. Patentability shall not be negatived by the manner in which the invention 
was made. 

The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 
USPQ 459 (1966), that are applied for establishing a background for determining 
obviousness under 35 U.S.C. 103(a) are summarized as follows: 

1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating 
obviousness or nonobviousness. 

Claims 1,5, 11, 14, and 40 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Kawasaki (reference AAA, IDS submitted with last response, 8/8/05) 
in view of Haala and Wang (reference AQ, IDS submitted with last response, 8/8/05). 

As to claim 1 , 
As to claims 1, 14, and 40, 
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A method comprising: 

-Generating a three dimensional model of an environment from range sensor 
information representing a height field for the environment; (Haala section 1 , pgs. 105- 
106, see especially Fig. 1 where part of a 3-D model derived from an airborne LIDAR 
range sensor is shown, where clearly this constitutes a "height field" for the urban 
environment) 

-Tracking orientation information of at least one image sensor in the environment with 
respect to the three-dimensional model in real-time; (Kawasaki, section 3.2 and 3.3 
(pages 14-15), where a vehicle is equipped with GPS is used, where this clearly 
teaches tracking orientation/position information of the at least one image sensor, note 
in section 3.3 it clearly states that a car using GPS and multiple cameras is used, and it 
states very clearly on page 14 in section 3.2 that the system uses a digital map, which is 
comparable to the three-dimensional model, see for example sections 4.1-4.2 on pages 
16-17, where the digital map is shown, where clearly tracking position with respect to 
the map is inherent because streaming video is shown in Figure 14, where such video 
can clearly be placed into the environment for matching purposes, see Figure 15 where 
the user has a streaming video that is put into the system and the user asks questions / 
queries the system concerning some feature in the real-time video stream, and clearly 
this is done in real-time, as in section 4.1, where (pages 16-17). Clearly, for the system 
to be able to maintain a correlation between vehicle location on the digital map and the 
input system, the tracking must be done in real-time, particularly since it uses streaming 
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video and allows real-time query and answer as noted above.)(Wang teaches that the 
user location is tracked in real-time - see page 5, section 3.1 ) 
-Projecting real-time video imagery information from the at least one image sensor onto 
the three dimensional model based on the tracked orientation information; and 
(Kawasaki page 17, section 4.2 - "All divisions are done automatically by projection the 
corresponding edge pattern obtained by the DP matching..." Clearly, also see section 
4.3, where the geometric model is constructed and the texture data is placed onto such 
a model, where the texture data is put (projected) onto a three-dimensional 
model.)(Wang clearly teaches real-time video visualization, see section 3.1 on page 5, 
and the video avatar is mapped onto the three-dimensional model; see page 7. The 
video image is back-projected onto a static head model using a projective texture 
matrix, where this clearly represents mapping the video imagery to the three- 
dimensional model. Note section 3.3.2, where the projective transformation is clearly 
described as being "...as if the camera is being replaced by a slide-projector which 
projects the images back onto the model.) 

-Visualizing the three-dimensional model with the projected real-time video imagery. 
(The system of Kawasaki clearly teaches that 3D maps can be made by this system in 
VRML, where the environment is visualized based on the texture data placed on the 
three-dimensional map (see Figure 1 9 for example))(Wang - Results are clearly 
visualized, as in Figure 5, where the video avatar representing a remote user is shown. 
The video image is projected onto the 3D head model, which clearly shows the required 
system.) 



Application/Control Number: 10/676,377 Page 10 

Art Unit: 2672 

The Kawasaki reference teaches a system that takes in video and matches it to a 
digital map. The reference does not expressly state whether or not the map is two or 
three-dimensional, but such maps are typically generated (at least in the United States) 
using government-provided DEM (digital elevation models) that are overlaid on existing 
street maps or the like, and thusly have at least some indication of altitude and/or height 
(examiner takes Official Notice on this topic as well), and it is notoriously well-known in 
the art as well as being obvious that such maps would and/or could be three- 
dimensional), and generates a final output result that includes a VRML map of the 
viewed area. Clearly, since the tracking is done in real-time (e.g. the real-time question- 
and-answer / query system, the use of "streaming" video which is well-known in the art 
to indicate real-time video), and the like. The reference does not expressly teach the 
use of 3D model generated from airborne telemetry, but such a 3D model is also 
equivalent to a 3D map. In any case, such representations are well known in the art. 
Further, as noted extensively above, the Kawasaki reference teaches that the textures 
and video are projectively mapped onto a three-dimensional model or other limitations. 

The Haala reference teaches the use of an airborne LIDAR platform that obtains 
three-dimensional height data and then overlays it with information from a 2-D GIS to 
form a three-dimensional representation of urban terrain. This system is a perfect 
example of sensor fusion to create the three-dimensional system database or 
geospatial database of Kawasaki that is required. Further, Haala clearly teaches the 
combination of DSM (Digital Surface Models) and existing 2D GIS databases on the 
geometry and usage of buildings in combination with digital maps, therefore the 
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combination of all of the above for the digital map of Kawasaki would be credible. 
Further, Haala illustrates another point, and that is that simply because a 3D model has 
been created, there is not necessarily video or texture data for the surfaces of the 
buildings, merely height information. 

The combination of the Kawasaki reference and the Haala reference is 
appropriate for several reasons; firstly, the Kawasaki reference does not cover height 
information per se, wherein the models are formed but the actual height information is 
never discussed per se, and the Haala reference allows for the inclusion of such data in 
the output VRML model; secondly, Haala provides a 3D model but without texture data 
of the buildings at street level, which Kawasaki clearly provides. The combination of 
these two references complement each other very clearly, and further allows Kawasaki 
to include data from DSMs, 2D GIS models, and many other data sources not 
specifically taught in the generic "digital maps" of Kawasaki. Further, for Kawasaki to 
accurately represent buildings in VRML, the system must have some source of height 
data, thus lending support to the previous contention that the "digital maps" of Kawasaki 
must prima facie include height data and/or be three-dimensional for at least the above 
reasons. 

Kawasaki provides a three-dimensional visualization capability, where textures 
are obtained and then mapped to the 3D models of the buildings. 

The Wang reference clearly teaches taking input, real-time video and mapping it 
to a virtual environment containing three-dimensional avatars and other elements of a 
3D model, see for example Figure 6 on page 1 1 , Figure 7 on page 12 particularly, 
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where the user can clearly navigate through a three-dimensional environment with the 
avatar, which clearly shows the idea of mapping real-time video to the three- 
dimensional environment as required by the claim. 

Kawasaki does not expressly teach that the textures are derived from the 
streaming video, but such capabilities would be obvious since the VRML output model 
is generated, and the system takes in real-time streaming video and allows the user to 
query the system concerning elements in the video stream to obtain information. That 
clearly requires that the data go into the system. It would be obvious that such data 
could supplant or augment that of the stored textures (for example, building surface 
under different lighting conditions and the like). Next, for visualization purposes such 
information would be useful, because the textures must be back-projected anyway, as 
in Wang, and Kawasaki is relatively silent on the details of the projection process. 
Specifically, Wang provides extensive implementation details for how such textures are 
mapped onto existing three-dimensional models. More specifically, Wang allows video 
imagery to be projected onto portions of three-dimensional models. Both the 
implementation and concept of Wang are important, the implementation because it 
specifically fills in methods for how Kawasaki would perform such transformations and 
projective mapping, and the concept of allow video to be a texture. This would therefore 
expressly teach the combination of video textures with the three-dimensional model with 
the textures of Kawasaki (and secondarily, of Haala). Therefore, for at least the above 
reasons, it would have been obvious to one of ordinary skill in the art at the time the 
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invention was to combine Kawasaki with Haala and then to use the projective mapping 
techniques and video textures of Wang in the resultant three-dimensional models. 

Applicant is further put on notice that the combination as modified as above 
would be operative and would not change the principles of operation of the device, nor 
do any of the references teach away. If applicant chooses to dispute this point, 
examiner will supply extremely thorough and detailed analyses of the references, 
various references and other external evidence, and the relevant case law in the 
resultant final rejection that will serve as the basis for the Examiner's Answer. 

As to claim 4, clearly the system of Kawasaki teaches the use of multiple 
cameras, as noted above. 

As to claim 5, the Haala reference clearly teaches different levels of detail in 
Figs. 2 and 3, and Fig. 1 clearly shows the grid used to hold the data after it is captured 
(pgs. 106-107). Specifically, in the conclusion on pg. 1 1 1 teaches that varying the level 
of detail is known in the art, e.g. "The level of detail, which can be reached for a 
reconstitution has to be examined depending on the density of measured laser points." 
Based on these statements, it is obvious that the points are projected onto a regular grid 
in any case (based on rejection to claim 1 above, where projection is taught) - the case 
of user-defined resolution is as taught above; the reference teaches that it would have 
been obvious to modify the reference to use multiple resolutions or levels of detail. 
Motivation and combination are taken from the parent claim. 

As to claim 1 1 , Kawasaki clearly teaches in section 3.1 the acquisition of depth 
from the video stream based on motion vectors and the like. Clearly, in Figure 19 on the 
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last page of the Kawasaki reference, the VRML viewer shows the image based on 
position, e.g. occluded portions because of depth are not shown; this is inherent in the 
functioning of the VRML viewer, since occluded areas are not shown; further, Wang 
clearly teaches navigating through a 3D landscape with the avatars, such that if an 
avatar is turned away from the viewer, the avatar's video face would not be visible, 
which teaches depth map coverage by occlusion as required. Any other differences 
would be obvious. 

As to claim 21 , the system of Kawasaki clearly utilizes GPS inputs as discussed 
in sections 3.2-3.3, pages 14-15. 

Claims 2-3, 7, 18, and 41 and 45-49 are rejected under 35 U.S.C. 103(a) as 
unpatentable over Kawasaki, Haala, and Wang as applied to claim 1 above, and further 
in view of Watanabe et al (US PGPub 2001/0005425). 

As to claims 2 and 41 , this is similar to claim 1 , the rejection to which is 
incorporated by reference. The main difference is the added limitation of 
-Wherein generating the three dimensional model comprises identifying a structure in 
the range sensor information; identifying different sections of the structure, selecting 
geometric primitives for the different sections of the structure based at least in part on 
input from a person regarding different shapes of the different sections, and parametric 
fitting of the geometric primitives to the range sensor information. 

Haala teaches in section 2 (page 106) that the generation of the three- 
dimensional model relies on the extraction of geometric primitives that correspond to 
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items like a roof of a building, and this data is of course from the range sensor. 
Specifically, the first sentence of section 2.1 (page 107) specifically states, "range 
image segmentation, which aims at dividing the object surface into patches that can be 
described parametrically ..." The work of Haala does not teach away from this, merely 
stating in the next line that their work is limited to a subset of that because that is all that 
is necessary, with multiple algorithms tested for use in their system. Haala further 
teaches that different sections of the structure are recognized based on the height of 
eaves, roofs and the like. 

The systems of Haala, Wang, and Kawasaki do not expressly teach where the 
geometric primitives are in fact chosen at least in part by input from a person. The 
system of Watanabe teaches that a plurality of images are input and a base three- 
dimensional object is shown, where the user then selects portions of the model that 
correspond to each other from the images and corrects the underlying model (see 
[0038-0042]). It would have been obvious to one of ordinary skill in the art at the time 
the invention was made to combine the system of Watanabe with that of Haala et al as 
above, because the system of Watanabe allows for the correction of three-dimensional 
models based on the various input images, where the user can clarify the relationships 
between different portions of the input image and correct the deficiencies, e.g. any 
potential gaps or areas where the correspondence is not quite correct, which would 
clearly benefit the system of Haala as combined with the other systems above. See as 
an example, Figure 3 of Watanabe, where buildings are classified into various types and 
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the user can manipulate them as required, where the user can select a model in Figure 
4, and various examples of geometries are shown in Figures 5-9. 

As to claims 3, 46, and 48, Watanabe clearly teaches that the primitives can 
have different types, such as rectangular solid, sphere, cylinder, and the like - see 
[0048], and it is well known in the art to use such elements, and clearly the user selects 
the primitive that best fits the model data, as in [0041], where the use parametric best 
fitting would be obvious in light of Haala as discussed above. A cube is clearly a type of 
rectangular solid, and a cuboid primitive Would be such as well. 

As to claim 7, Haala is a LIDAR system (specifically, see pg. 106, section 1, or 
the tagline on Fig. 1 , "DSM measured by airborne laser scanning." Motivation and 
combination are incorporated by reference from the parent claim. 

As to claim 18, the first part of the claim is identical to claim 14, the rejection to 
which is incorporated by reference. Further, the latter part of the claim is identical to 
claim 2 above, which is incorporated by reference as well. 

As to claims 45, 47, and 49, examiner contends that Watanabe clearly teaches 
that such primitives (cuboid, sphere, and the like) are possible for such applications, 
and thusly would be covered, since 'including' is open ended and does not require all 
elements be present. Further Haala teaches that such structures can be analyzed and 
segmented without ground plan information. 



Claim 4 is rejected under 35 U.S.C. 103(a) as being unpatentable over Kawasaki, 
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Haala, Wang, and Watanabe as applied to claim 2 above, and further in view of Arpa et 
al (US 2003/0085992 A1). 

As to claim 4, references Kawasaki, Wang, Haala, and Watanabe do not 
expressly teach this limitation, while Kawasaki does teach the use of multiple cameras. 
Reference Arpa teaches this limitation, specifically in Fig. 1 where multiple cameras 
(108, 1 10, 1 14) are shown being linked to an image processor, where the system in Fig. 
2 then takes all the video inputs and projects them onto a 3D model using 3D model 
generator 210; the results are shown in for examples Figs. 7 and 8 where an object that 
is moving is tracked (see [0026-0028]). Clearly, the system of Arpa projects the results 
of multiple video cameras onto a known three-dimensional model of a scene. The 
specific limitation of "refining the three-dimensional model based on object surfaces" is 
clearly performed by Haala, as for example in Figs. 10-13, where initial reconstruction of 
the scene is shown and the adjusted or corrected version is shown (which is 
comparable to the recited "further ... refining" recjted in the claim). 

Clearly, It would have been obvious to one having ordinary skill in the art at the 
time the invention was made to combine the systems of Kawasaki, Haala, and Wang 
with that of Arpa, as the system of Kawasaki teaches the use of one camera and 
multiple cameras under various circumstances, but does not discuss if the cameras 
overlap or similar; the system of Arpa would prima facie enhance the capability of that 
system by allowing it handle multiple cameras (Note that claims 15 and 20 are very 
similar to / obvious variants of one another and are covered by the above rejection). 
Arpa is an analogous art. 
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Claims 10, 15, 16, and 17 are rejected under 35 U.S.C. 103(a) as unpatentable 
over Kawasaki, Haala, and Wang as applied to claims 1 and 14 above, and further in 
view of Arpa. 

As to claim 10, the system of Arpa clearly covers the scenario of both multiple 
image sensors as shown above, and claim 1 itself teaches the application of real-time 
video imagery information. The limitation of multiple video streams being projected onto 
the three-dimensional variant is taught by Arpa, where all the views of the cameras are 
incorporated into the 3D model so that a user can choose between desired views with 
respect to the structural model, for example Figs. 1 and 2. Motivation and combination 
are taken from the rejection to claim 4 above. 

As to claim 15, reference Arpa teaches this limitation, specifically in Fig. 1 where 
multiple cameras (108, 1 10, 1 14) are shown being linked to an image processor, where 
the system in Fig. 2 then takes all the video inputs and projects them onto a 3D model 
using 3D model generator 210; the results are shown in for examples Figs. 7 and 8 
where an object that is moving is tracked (see [0026-0028]). Clearly, the system of 
Arpa projects the results of multiple video cameras onto a known three-dimensional 
model of a scene. The specific limitation of "refining the three-dimensional model based 
on object surfaces" is clearly performed by Haala, as for example in Figs. 10-13, where 
initial reconstruction of the scene is shown and the adjusted or corrected version is 
shown (which is comparable to the recited "further ... refining" recited in the claim). 
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Clearly, It would have been obvious to one having ordinary skill in the art at the 
time the invention was made to combine the systems of Kawasaki, Haala, and Wang 
with that of Arpa, as the system of Kawasaki teaches the use of one camera and 
multiple cameras under various circumstances, but does not discuss if the cameras 
overlap or similar; the system of Arpa would prima facie enhance the capability of that 
system by allowing it handle multiple cameras (Note that claims 15 and 20 are very 
similar to / obvious variants of one another and are covered by the above rejection). 
Arpa is an analogous art. 

As to claim 16, Arpa clearly teaches in [0036] that such video can recorded, 
rewound, replayed, and the like, thusly teaching the use of pre-recorded video imagery. 

As to claim 17, Arpa teaches in Fig. 8 the use of a 'synthetic' view to show the 
location of a moving object, which is equivalent to the recited limitation of computing the 
view of the virtual camera from a viewpoint not that of one of the image sensor; see 
[0047] specifically for the details on how the virtual camera works. Motivation and 
combination is taken from the rejection to claim 15 and incorporated by reference. 

As to claim 20, this is a duplicate of claim 15, the rejection to which is 
incorporated by reference, where Arpa clearly teaches refining the image. 

Claim 6 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Kawasaki, Haala, and Wang as applied to claim 1 above, and further in view of 
Weinhaus et al (Weinhaus, F. and V. Devarajan. "Texture Mapping 3D Models of Real- 
World Scenes.") 
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As to claim 6, references Kawasaki, Haala, and Wang do not expressly teach this 
limitation except that Haala discusses filling gaps in the images using parametric fitting, 
while reference Weinhaus teaches the limitations of tessellation and hole filling on pg. 
349 where it is stated that terrain elevation tiles are tessellated into triangular meshes 
and on pg. 346 where methods using point-projection methods to fill holes. Therefore, it 
would have been obvious to modify the combination of Kawasaki and Haala as modified 
by Wang to use the techniques recited in Weinhaus to further refine and correct the 
three-dimensional models generated during the process, given that in pgs. 340-347 
several different techniques are given for performing texture mapping on height-based 
data sets using projective techniques. 

Claim 19 is rejected under 35 U.S.C. 103(a) as unpatentable over Kawasaki, 
Haala, Wang, and Watanabe as applied to claim 1 above, and further in view of Bar. 

As to claim 19, it is substantially the same as claim 6, with the additional 
limitation of using a user-defined resolution is met in the rejection to claim 6 above, 
which uses only the Kumar and Haala references. Motivation and combination is taken 
from claim 6 above and incorporated in its entirety by reference. 

Claim 8 is rejected under 35 U.S.C. 103(a) as unpatentable over Kawasaki, 
Haala, and Wang as applied to claim 1 above, and further in view of Jiang et al 
(reference All in the first IDS filed on 4 March 2004, B. Jiang, U. Neumann, "Extendible 
Tracking by Line Auto-Calibration.") 
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As to claim 8, Kawasaki clearly teaches tracking camera orientation using line 
and edge orientation of buildings so that it can be fitted to the final model. Likewise, the 
GPS systems of Kawasaki clearly allow the user to extract both position and orientation 
information anyway (bearing is derivative of positional data over time), which is tracked 
on the maps anyway, which would prima facie allow the user to determine orientation, 
and clearly Kawasaki teaches this limitation anyway (section 3.1, page 14), and Haala 
also suggests matching 2D GIS information and/or other sources of data on ground 
structures to LIDAR data for correction of errors in various places. All of the above fairly 
suggest estimating the camera pose from features in the images. 

In the interests of expediting prosecution, reference Jiang is brought in. Jiang 
clearly teaches in Figure 2 and sections 2 and 3 the use of camera tracking and line 
auto-calibration, where the system allows the extension of tracking using optical 
measurements such that it can use lines to navigate to track its location and particularly 
to determine camera pose (see Figures 5 and 6 on page 5 of the document, captions 
'camera pose was estimated both from pre-calibrated landmarks and auto-calibrated 
line features'). 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to provide for a method of explicitly tracking camera orientation / 
pose as provided by Jiang, because such methods inherently improve the data set by 
allowing tracking of orientation (and location) in regions where GPS may not work (e.g. 
urban canyon situations) and further by allowing explicit tracking of camera position and 
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pose via optical features as set forth therein. Therefore, such a modification and/or 
inclusion would be beneficial for at least the above reasons and proper. 

As to claim 9, clearly Kawasaki teaches the use of GPS sensors in sections 3.2- 
33, on pages 14-15. 

Claims 13 and 23 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Kawasaki, Haala, and Wang as applied to claim 1 above, and further in view of 
Pryor et al (US 2004/0046736 A1)('Pryor'). 

As to claims 13 and 23, references Kawasaki, Haala, and Wang do not expressly 
teach this limitation, but Wang does teach a VR type interface, where users can 
navigate avatars with video textures and the like. Reference Pryor teaches the use of a 
virtual reality type user interface wherein large images are projected onto a screen, 
such as a wall screen [0043, 0056] and that many input devices, such as a head tracker 
are well known in the art and can be used with the invention of Pryor [0009, 0067, 0083- 
0085, particularly 0105 where it says that the user's head can be tracked for purposes 
of motion determination]. It would have been obvious to one having ordinary skill in the 
art at the time the invention was made to combine the systems of Kawasaki, Haala, and 
Wang with that of Pryor, given that in [0056] it is taught that using large screens makes 
it easier to interact with imagery and obviously this would facilitate data visualization 
and similar with the combination of Kawasaki, Haala, and Wang, particularly with Wang. 
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Claims 45, 47, and 49 are rejected under 35 U.S.C. 103(a) as unpatentable over 
Kawasaki, Haala, Wang, and Watanabe as applied to claims 3, 46, and 48 above, and 
further in view of Chen et al (US PGPub 2003/0147553). 

References Kawasaki, Haala, and Wang do not expressly teach this limitation. 
Watanabe clearly teaches the use of cylinder, cuboid, and similar primitives as 
discussed in the previous paragraph. While examiner maintains that since the above 
references teach all the limitations (e.g. Watanabe provides all the primitives required 
and Haala can function without them), in order to expedite prosecution, the Chen 
reference is brought in. 

The Chen reference teaches the use of a roof primitive for reconstructing 
structural data, which clearly would apply to the system of Haala and Kawasaki. The 
roof-primitive [0018] and [0025-0032] clearly teach how images can be split when there 
is a shared edge, which would clearly facilitate the process of segmenting out buildings 
and the like in Haala. Therefore, for at least the above reasons, it would have been 
obvious to one of ordinary skill in the art to use the methods of Chen to avoid occluded 
corners and other defects [0017-0018], where the system is semi-automatic, therefore 
allowing the user to select various elements of the process as required by claim 2 and 
would improve the refinement capabilities of Watanabe. 
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Conclusion 



Applicant's submission of an information disclosure statement under 37 CFR 
1 .97(c) with the fee set forth in 37 CFR 1 . 1 7(p) on 8 August 2005 prompted the new 
ground(s) of rejection presented in this Office action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Eric V. Woods whose telephone number is 571-272- 
7775. The examiner can normally be reached on M-F 7:30-4:30 alternate Fridays off. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Michael Razavi can be reached on 571-272-7664. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-21 7-91 97 (toll-free). 



Eric Woods 




November 15, 2005 



